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DETAILED ACTION 

1. This action is responsive to communications: Request for Reconsideration (hereinafter the Request) 
filed 9/20/2004, to the original application filed 2/6/2002, drawings filed 3/25/2002. The present application is a 
continuation (filed under 37 CFR 1.53(b)) of parent case 08/900,42 1 (now abandoned). Said parent case filed 
7/25/1997. 

2. Claims 1, 3-20 remain rejected under 35 U.S.C. 103(a) as being unpatentable over various combinations 
of Mendelson, Baumgartner, Shibata, Katseff, andDurward 

3. Claims 1, 3-20 are pending. Claims 1, 14, 16, 19, 20 are independent claims. 

Claim Rejections - 35 USC §103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the 
differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall 
not be negatived by the manner in which the invention was made. 

5. Claims 1, 8-10, 12-13, 16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Mendelson et al. (hereinafter Mendelson), U.S. Patent No. 5,754,783 issued May 1998. 

In regard to independent claim 1, Mendelson teaches interleaving (i.e. synchronizing) timed primary 
and untimed secondary data streams at a server, said resulting stream(s) eventually transported to clients for 
display (Mendelson Abstract, column 4 lines 18-22, column 8 lines 15-20). Mendelson defines timed primary 
content data as games, movies, audio music, etc., acquired and stored in a storage device (hard drive) on said 
server (Mendelson column 1 lines 13-22, column 6 lines 8-1 1, Figure 4 item 421). Mendelson defines untimed 
secondary content as video text, graphic images, private data such as files, databases, or stuffing bits, etc. 
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(Mendelson column 1 lines 24-28, column 3 lines 50-54, column 6 lines 23-26, Figure 4 item 414). It is noted 
that any number of data streams can be utilized (Mendelson column 4 lines 54-58). 

Mendelson does not specifically disclose "motion" type data. However, Mendelson's secondary content 
(video text/images etc.) can at least be interpreted as suggestive of motion data since it is displayed as a 
video/movie type stream. In addition, Mendelson's secondary content can also comprise "padding", null/stuffing 
data, interleaved via a stuffing buffer, helping to preserve the quality (i.e. motion - wander, jitter, etc.) of the 
timed primary data (Mendelson column 2 lines 50-67, column 3 lines 9-20, 50-54, column 8 lines 14-17, Figure 
4 item 440). It would have been obvious to one of ordinary skill in the art at the time of the invention to interpret 
Mendelson's secondary data as a type of "motion" data, providing the benefit of controlling/minimizing possible 
timed motion related problems (i.e. wandering, jittering, etc.) of the primary content (compare the above two 
paragraphs with claim 1 "A method of synchronizing asynchronous time-based and motion data in a system in 
which the time-based data and the motion data are transmitted by a server over a network to a client, the 
method comprising", and "retrieving a time-based data stream and a motion data stream at the server... ") 

Mendelson teaches receiving various time-based and motion data at a server, as explained above. In 
addition, since Mendelson discloses typical timed and untimed content within a frames environment (Mendelson 
column 1 lines 17-18, 24-25), combined with Mendelson's teaching of using the MPEG standard (typically 
applied to MPEG files, which are frame-based) (Mendelson column 4 lines 48-52), Mendelson's data streams 
can be interpreted as frame based (compare with claim 1 "... each stream comprising frames of data"). 

Mendelson teaches variable buffering (at a server) by measuring the bit rate stream, and buffers via the 
use of variable usage (interleaving) of primary, secondary and stuffing content buffers (Mendelson column 6 
lines 62-67, column 8 lines 6-20, Figure 4 items 440, 450, 460). It is noted that each type of content has its own 
buffer, and are regulated accordingly. It is noted that Mendelson's resulting data stream can be interpreted as 
comprising at least two interleaved streams (now custom synchronized) of various data types. In addition, 
multiple elementary streams within a transport stream can be utilized (Mendelson column 4 lines 54-63, column 
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6 lines 4-8) (compare with claim 1 "variable buffering one of the time-based data stream and the motion data 
stream at the server to produce two streams having synchronized frames"). 

Mendelson teaches transport of a final (custom synchronized) stream(s) to customers (clients) for 
eventual playback (Mendelson column 6 lines 4-8, column 8 lines 10-12; compare with claim 1 "using the 
synchronized frames at the client for playback of synchronized motion and time-based data to a user"). 

In regard to dependent claim 8, Mendelson teaches primary content (video, audio data, etc.) as well as 
secondary content (video text, images, files, database data, along with "stuffing" data, etc.) as explained in the 
rejection of claim 1. As the resulting interleaved stream(s) is played back on a client, any combination of 
foreground/background combination can be typically specified, (i.e. the primary data (i.e. a movie scene) can be 
run as background to the secondary/motion data controlling it, or run as background to any related database files 
in the foreground, etc). 

In regard to dependent claims 9, 10, Mendelson teaches buffering/processing occurring at the same 
time the program is being transported to customer equipment (Mendelson column 8 lines 9-12; compare with 
claim 9). 

Mendelson teaches audio (i.e. voice) data (Mendelson column 4 lines 57-59; compare with claim 10). 

In regard to dependent claims 12, 13, Mendelson teaches buffering of streams comprising buffering of 
primary and secondary content (said primary content can be audio data, Mendelson column 4 lines 57-59), along 
with necessary interleaved motion type data (secondary/stuffing data), producing a sequenced synchronized 
stream regarding constant verses variable bit stream rate (see also rejection of claim 1, also Mendelson column 6 
lines 62-67). 

In regard to independent claim 16, claim 16 reflects the apparatus used to perform the methods 
comprising computer readable instructions as claimed in claim 1, and is rejected along the same rationale. 
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6. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable Mendelson as applied to claim 1 
above, and further in view of Baumgartner et al. (hereinafter Baumgartner), U.S. Patent No. 5,642,171 
issued June 1997. . 

In regard to dependent claim 3, Mendelson teaches buffering based on a calculated difference (delay) 
between a variable bit stream, and a constant bit stream (Mendelson column 6 lines 62-67). Mendelson does not 
specifically teach a difference between the delay for motion and time-based data stream to determine buffering 
for a faster stream. However, Baumgartner teaches a method whereby a current video frame number is 
subtracted from a current audio frame number to determine if the audio is too far ahead of the video 
(Baumgartner column 13 lines 60-67, column 14 lines 1-2). It would have been obvious to one of ordinary skill 
in the art at the time of the invention to apply the difference calculation method of Baumgartner to Mendelson' s 
stream analysis, providing Mendelson the benefit of more accurate time synchronization to the server buffer 
method disclosed by Mendelson. 

7. Claims 4, 14-15, 18 is rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, and 
further in view of Shibata, Y., Media synchronization protocols for packet audio-video system on 
multimedia information networks (hereinafter Shibata), IEEE; January 3-6, 1995 pp.594-601. 

In regard to dependent claim 4, Mendelson does not specifically teach transfer of data values for a 
frame occuring only for a frame that has changed since a last frame was transmitted. However, Shibata discloses 
a method whereby audio data is sent from the video server to the client station only during a c *talk spurt", with 
constant frame rates occurring during periods of audio silence (Shibata p.597, section 4.3 "Silence detected 
synchronization"). It would have been obvious to one of ordinary skill in the art at the time of the invention to 
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apply Shibata to Mendelson's stream interleaving, providing Mendelson the benefit of file size and transmission 
reduction (streamlining) of the audio/video data streams. 

In regard to independent claim 14, claim 14 incorporates substantially similar subject matter as 
claimed in claim 1, and in further view of the following, is rejected along the same rationale. 

Mendelson teaches an embodiment using streams associated with "data channels" (Mendelson column 
6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), said header typically used to describe 
data within transport stream data (i.e. each 188 byte packet) (Mendelson column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach transfer of data values (channels) for a occuring only for changed 
data values. However, Shibata discloses a method whereby audio data is sent from the video server to the client 
station only during a "talk spurt", with constant frame rates occurring during periods of audio silence (i.e. a talk 
spurt changes the data values) (Shibata p.597, section 4.3 "Silence detected synchronization"). It would have 
been obvious to one of ordinary skill in the art at the time of the invention to apply Shibata to Mendelson's 
stream interleaving, providing Mendelson the benefit of file size and transmission reduction (streamlining) of 
the audio/video data streams. It is to be noted that in order to adjust frame rates as above, the server must know 
that a change in the data has occurred, which is typically achieved by comparing new data (the talk spurt) with 
the last recognized data value (previously recognized as empty - reflecting a period of silence). 

In regard to dependent claim 15, Mendelson teaches an embodiment using streams associated with 
data channels (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

In regard to dependent claim 18, Mendelson teaches an embodiment using streams associated with 
"data channels" (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
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said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach transfer of data values (channels) for a occuring only for changed 
data values. However, Shibata discloses a method whereby audio data is sent from the video server to the client 
station only during a "talk spurt", with constant frame rates occurring during periods of audio silence (i.e. a talk 
spurt changes the data values) (Shibata p. 597, section 4.3 "Silence detected synchronization"). It would have 
been obvious to one of ordinary skill in the art at the time of the invention to apply Shibata to Mendelson's 
stream interleaving, providing Mendelson the benefit of file size and transmission reduction (streamlining) of 
the audio/video data streams. It is to be noted that in order to adjust frame rates as above, the server must know 
that a change in the data has occurred, which is typically achieved by comparing new data (the talk spurt) with 
the last recognized data value (previously recognized as empty - reflecting a period of silence). 

8. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, as applied to 
claims 1, above, and further in view of Katseff et al. (hereinafter Katseff), U.S. Patent No. 5,822,537 issued 
October 1998. 

In regard to dependent claim 5, Mendelson teaches a networking environment (Mendelson column 4 
lines 28-33). Mendelson does not specifically teach using the Internet for distribution. However, Katseff teaches 
an internet as an exanple of suggested types of networks that can be used (Katseff, column 3 lines 58-63). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply Katseff to 
Mendelson, providing Mendelson the benefit of a widespread and familiar network. 
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9. Claims 6, 7, 11, 17, 19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Mendelson, 
and further in view of Durward et al. (hereinafter Durward), U.S. Patent No. 9,950,202 issued September 
1999. 

In regard to dependent claim 6, Mendelson teaches secondary data as stuffing data, or as private data 
such as files or databases (see rejection of claim 1, also Mendelson column 6 lines 24-26). Mendelson does not 
specifically teach said motion data mapped to control movement of a virtual figure displayed in a scene at the 
client. However, Durward teaches a method whereby updated positional data from a person's head position 
sensor is mapped and used to determine the position of a virtual being defined for that user, communicating 
graphical data to the user via database data (Durward, column 6 lines 29-32, 46-52, column 7 lines 12-20). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply the virtual 
position database data of Durward to Mendelson's secondary database data, providing Mendelson increased 
spatial imagery to the multimedia stream 

In regard to dependent claim 7, Mendelson does not specifically teach using a body suit for 
generating motion data. However, Durward teaches a method of sensing position data with the use of a body suit 
(Durward, column 3 line 25-33). It would have been obvious to one of ordinary skill in the art at the time of the 
invention to apply Durward to Mendelson, providing Mendelson the benefit of three dimensional physical 
contour mapping, providing increased spatial image accuracy to Mendelson's video stream method. 

In regard to dependent claim 11, Mendelson teaches an embodiment using streams associated with 
data channels (Mendelson column 6 lines l-8).Mendelson also teaches a header packet (a descriptor packet), 
said header typically used to describe data within transport stream data (i.e. each 188 byte packet) (Mendelson 
column 4 lines 47-53, Figure 2). 

Mendelson does not specifically teach that a client may join in progress a multicasting of a stream 
However, Durward teaches virtual reality presentations where a number of contributing user's data is collected 
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and analyzed accordingly, so a to facilitate multicasting (and interaction between figures) of the virtual world to 
various users (Durward Figures 4-6, 7 column 8 lines 5-15). Durward also teaches users logging on to a system 
via modems, said users data is shared with other users and continuously updated so as to create a virtual reality 
world (Durwald, column 8 lines 22-24, 41-64, Figure 1). It would have been obvious to one of ordinary skill in 
the art at the time of the invention to apply Durward to Mendelson, providing Mendelson the benefit of 
multicasting associated with synchronization of streams. 

In regard to dependent claim 17, Mendelson does not specifically teach an apparatus whereby 
synchronized motion/time-based data is milticasted to clients in a network. However, Durward teaches an 
apparatus whereby users log on to a system via modems, said users data is shared with other users and 
continuously updated so as to create a virtual reality world (Durward, column 8 lines 22-24, 41-64, Figure 1). It 
would have been obvious to one of ordinary skill in the art at the time of the invention to apply the data sharing 
method of Durward to Mendelson, providing Mendelson the benefit of continuous updating and increased 
uniform user awareness in a virtual environment. 

In regard to independent claim 19, claim 19 incorporates substantially similar subject matter as 
claimed in claim 1, and in further view of the following, is rejected along the same rationale. 

Mendelson teaches secondary data as stuffing data, or as private data such as files or databases (see 
rejection of claim 1, also Mendelson column 6 lines 24-26). Mendelson does not specifically teach said motion 
data mapped to control movement of a virtual figure displayed in a scene at the client, along with 
synchronization. However, Durward teaches a method whereby updated positional data from a person's head 
position sensor is mapped and used to determine the position of a virtual being defined for that user, 
communicating graphical data to the user via database data (Durward, column 6 lines 29-32, 46-52, column 7 
lines 12-20). Since the playback of Mendelson's resulting stream synchronizes the secondary data with the 
primary content, the primary data (i.e. video) can be synchronized with added secondary database data (related 
to a virtual figure). It would have been obvious to one of ordinary skill in the art at the time of the invention to 



Application/Control Number: 10/068,829 
Art Unit: 2176 



Page 10 



apply the virtual position database data of Durward to Mendelson's secondary database data, providing 
Mendelson increased spatial imagery and desired synchronization to the resulting multimedia stream. 

10. Claim 20 is rejected under 35 U.S.C 103(a) as being unpatentable over Mendelson, Baumgartner, 
and Durward. 

In regard to independent claim 20, claim 20 incorporates substantially similar subject matter as 
claimed in claims 1, 3, 10, 1 1, and is rejected along the same rationale. 

Response to Arguments 

1 1 . Applicant's arguments filed 9/20/2004 have been fully and carefully considered but they are not 
persuasive. 

Applicant argues on pages 8 to top half of page 10 of the Request that Mendelson does not specifically 
teach "variably buffering one of the time-based data stream and the motion data stream at the server to produce 
two streams having synchronized frames" (see instant claim 1). Applicant asserts that Mendelson teaches 
constant bit rates, instead of variable buffering, as presented above. 

It is respectfully noted that Mendelson (i.e. Figure 4) clearly shows analyzing and synchronizing data at 
a server. The examiner interprets the quoted limitation to mean In addition, the instant rejection shows two 
examples of how this can be achieved. Mendelson teaches variable buffering by measuring the bit rate stream 
and buffers via the use of variable usage (interleaving) of primary, secondary and stuffing content buffers. 
Please note that each type of content has its own buffer, and are regulated accordingly (Mendelson column 6 
lines 62-67, column 8 lines 6-20, Figure 4 especially items 440, 450, 460). It is noted that Mendelson's resulting 
data stream can be interpreted as comprising "at least two interleaved streams" (now custom synchronized) of 
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various data types, said interleaved file subsequently sent to a client for viewing. The recited portion of instant 
claim 1 does not limit the examiner from this interpretation. Mendelson's various streams are variable buffered 
as needed to produce a final synchronized file. Also note that Mendelson teaches that based upon a difference 
calculation, selective interleaving of content is applied from all three buffers (Mendelson column 8 lines 8-17). 

In addition, multiple elementary streams within a transport stream can be utilized (Mendelson column 4 
lines 54-63, see also rejection of instant claim 1). Alternatively, instead of (and/or in addition to) the above, a 
transport stream can include one or more separately identified elementary streams. The video and audio content 
can be two separately identifiable streams, with timed ancillary data, such as closed captioning (CC), transported 
as a third stream In this fashion, Mendelson processes and variably synchronizes all of the streams (via CC, 
etc.) at its server, then transports the final streams accordingly. 

Applicants arguments regarding claim 16 (page 10 of the Request) are similar to those presented above, 
therefore the examiner's rebuttal applies here as well. 

Applicant argues on pages 1 1 - 1 3 of the Request that the cited references do not teach the limitations of 
claims 3, 4, 5, 14, and 18. It is noted that Mendelson is relied upon to teach variable buffering, etc. as explained 
above. Regarding claim 3, Mendelson teaches buffering based on a calculated difference (delay) between a 
variable bit stream, and a constant bit stream (Mendelson column 6 lines 62-67). It is noted that since one of the 
bit streams is constant, it can be determined that the delay for this stream is zero. Mendelson does not 
specifically teach a difference between the delay for motion and time-based data stream to determine buffering 
"for a faster stream". However, Baumgartner teaches a method whereby a current video frame number is 
subtracted from a current audio frame number to determine if the audio is too far ahead of the video (in this case 
it is determined that audio is faster then video, the question is by how much faster) (Baumgartner column 13 
lines 60-67, column 14 lines 1-2). Regarding claim 4, Mendelson does not teach transfer of data values for a 
frame "occurring only for a frame that has changed since a last frame was transmitted". However, Shibata 
discloses a method whereby audio data is sent from the video server to the client station only during a "talk 
spurt", with constant frame rates occurring during periods of audio silence. In order to transfer data, Shibata 
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must differentiate between periods of inactivity, therefore a c; talk spurt" after a period of silence reflects a clear 
change in frame data, therefore signaling Shibata to transfer data accordingly. Regarding claim 14, Applicant 
asserts that the examiner does not point out in Mendelson where it teaches "storing a last data value for each 
channel in each frame transmitted over the network". The examiner interprets this limitation in the context of a 
streaming channel. In the case of interleaved data, as each frame is sequentially transmitted, the frame number 
associated with each frame can be "the last data value" (within the collection of frames so far transmitted), until 
arrival of the next frame, etc.. Regarding claim 5, Katseff is applied by the examiner to teach the Internet. 

Regarding Applicant's arguments on pages 14-15, Mendelson is relied upon to teach variable buffering, 
etc. as explained above. Durward is used to teach data related to a virtual figure, said data can be applied to 
Mendelson's secondary (i.e. timing or sequencing) data. Mendelson teaches synchronization using secondary 
data. The combination of Mendelson and Durward produces synchronization of video, audio and extra content 
streams (i.e. virtual reality data), accordingly. 

Applicant's arguments on pages 15-16 of the Request (regarding claim 20) are similar to issues 
previously presented. The examiner's previous rebuttals apply here as well. 

i 

Conclusion 

12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth 
in37CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the 
mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this 
final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 . 1 36(a) will be calculated from the mailing date of the advisory action. In no 
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event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this 
final action. 

13. Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to William L. Bashore whose telephone number is (571) 272-4088. The examiner can normally be 
reached on 1 1 :30am - 8:00pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Joseph Feild 
can be reached on (571) 272-4090. The fax phone number for the organization where this application or 
proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system Status information for published applications may be obtained from 
either Private PAIR or Public PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair-direct.usp to.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217- 
9197 (toll-free). 




William L. Bashore 
Patent Examiner, AU 2176 
November 23, 2004 



